Patch guide method and program

ABSTRACT

Disclosed is a patch guide method including acquiring a 3-dimensional scan model including the head of an object by using a depth camera, by a computer, acquiring a 3-dimensional brain MRI model of the object, matching the scan model and the brain MRI model to acquire a matched model, acquiring an image captured by photographing the head of the object by using the depth camera, and matching one location of the captured image and one location on the matched model.

CROSS-REFERENCE TO RELATED APPLICATIONS

A claim for priority under 35 U.S.C. § 119 is made to Korean PatentApplication No. 10-2017-0108056 filed Aug. 25, 2017, in the KoreanIntellectual Property Office, the entire contents of which are herebyincorporated by reference.

BACKGROUND

Embodiments of the inventive concept described herein relate to a patchguide method and a program.

An magnetic resonance imaging (MRI) system is a device that acquires animage for a tomographic portion of an object by expressing an intensityof a magnetic resonance (MR) signal for a radio frequency (RF) signalgenerated by a magnetic field of a specific intensity in contrast. Forexample, if an RF signal that resonates only specific atomic nuclei (forexample, hydrogen atomic nuclei) is instantaneously irradiated after anobject is laid in a strong magnetic field and the is stopped, an MRsignal is emitted from the specific atomic nuclei, and the MRI systemmay receive the MR signal and acquire an MR image. The MR signal refersto an RF signal radiated from the object. The intensity of the MR signalmay be determined by the concentration of atoms (for example, hydrogen)included in the object, a relaxation time T1, a relaxation time T2, andblood flows.

The MRI system includes features that are different from those of otherimaging devices. Unlike the imaging devices, such as a computerizedtomography (CT) device, in which acquisition of an image depends on adirection of detection hardware, the MRI system may acquire a 2D imageor a 3D volume image oriented toward an arbitrary point. Further, unlikea CT device, an X-ray device, a positron emission tomography (PET)device, and a single photon emission computed tomography (SPECT) device,the MRI system does not expose a radioactive ray to an object and aninspector and may acquire a neurological image, an intravascular image,a musculoskeletal image, and an oncologic image, in which it isimportant to clearly describe abnormal tissues by acquiring an imagehaving a high soft tissue contrast.

An electrical brain stimulation refers to a method of finally applying acurrent to a brain by attaching an electrode to an inside and an outsideof the head and allowing a current to flow to the electrode. Anelectrical brain stimulation is a non-invasive treatment method that maybe simply performed, and may be widely used to treat various braindiseases according to the location at which a stimulus is applied andthe type of the stimulus.

Further, an EEG (electroencephalogram) that may measure an electricalactivity according to the activity of the brain of an object is alsowidely used in the neurological and neuropsychiatric departments.

Both the electrical brain stimulation and the EEG brain wave inspectionare non-invasive inspection and treatment methods, and may be simplyperformed. However, because the brain structures of the brains and theshapes of the heads of people are different, it is difficult to attach apatch to an accurate location to perform a treatment by the doctor, andthus it is required to develop a patch guide method that reflects theshapes of the heads of people.

SUMMARY

The inventive concept provides a patch guide method and a program.

The technical objects of the inventive concept are not limited to theabove-mentioned ones, and the other unmentioned technical objects willbecome apparent to those skilled in the art from the followingdescription.

In accordance with an aspect of the inventive concept, there is provideda patch guide method including acquiring a 3-dimensional scan modelincluding the head of an object by using a depth camera, by a computer,acquiring a 3-dimensional brain MRI model of the object, matching thescan model and the brain MRI model to acquire a matched model, acquiringan image captured by photographing the head of the object by using thedepth camera, and matching one location of the captured image and onelocation on the matched model.

The patch guide method may further include recognizing at least onepatch included in the captured image, determining a location of therecognized patch in the captured image, and acquiring a location of thematched model corresponding to the determined location of the patch.

The matching of the scan model and the brain MRI model may includecalculating facial features of the scan model and the brain MRI model,and matching the scan model and the brain MRI model by using the facialfeatures of the scan model and the brain MRI model.

The calculating of the facial feature of the scan model may includeacquiring a color image and a depth image including the head of theobject, calculating the facial feature of the object by using the colorimage, and calculating a 3-dimensional location of the facial feature ofthe object by using the depth image.

The patch guide method may further include displaying an image thatguides a location of the patch, at which the head of the object is to beattached, in the captured image, and the displaying of the image mayinclude determining a location, at which the patch is to be attached, onthe matched model, and displaying a location corresponding to thedetermined image in the captured image.

The patch guide method may further include recognizing a patch in thecaptured image, guiding a movement direction of the recognized patch,and determining whether the recognized patch is attached at thedetermined location.

The acquiring of the 3-dimensional brain MRI model of the object mayinclude acquiring a brain MRI image of the object, and generating a3-dimensional brain map of the object, by which a delivery process of anelectrical stimulus for the brain of the object is simulated, based onproperties of a plurality of areas included in the brain MRI image, andthe patch guide method may include determining a location of the patch,at which the head of the object is to be attached, by using the3-dimensional brain map.

The generating of the 3-dimensional brain map may include generating a3-dimensional solid image constituted by a plurality of meshes, by whicha delivery process of an electrical stimulus for the brain of the objectis simulated.

The determining of the location of the patch may include acquiring anobject of using the patch, simulating a process of delivering anelectrical stimulus to the brain of the object according to a location,at which the patch is attached to the head of the object, anddetermining the location of the patch by using the acquired objectiveand the simulation result.

In accordance with another aspect of the inventive concept, there isprovided a computer readable computer program recorded in a recordingmedium, which is coupled to a computer that is hardware to perform themethod.

The other detailed items of the inventive concept are described andillustrated in the specification and the drawings.

BRIEF DESCRIPTION OF THE FIGURES

The above and other objects and features will become apparent from thefollowing description with reference to the following figures, whereinlike reference numerals refer to like parts throughout the variousfigures unless otherwise specified, and wherein:

FIG. 1 is a flowchart illustrating a method for generating a3-dimensional brain map according to an embodiment;

FIG. 2 is a flowchart illustrating a method for generating a3-dimensional brain map and performing simulation according to anembodiment;

FIG. 3 is a view illustrating a result of segmenting brain MRI images;

FIG. 4 is a view illustrating an example of a computer performsconnected component-based noise rejection method;

FIG. 5 is a view illustrating an example of a post-processing methodusing hole rejection;

FIG. 6 is a view illustrating an example of a 3-dimensional brain imagegenerated from a brain MRI image of an object;

FIG. 7 is a view illustrating an example of a diffusion tensor image;

FIG. 8 is a view illustrating an example of a simulation result;

FIG. 9 is a flowchart illustrating a TMS stimulus method according to anembodiment;

FIG. 10 is a view illustrating an example of a TMS treatment method;

FIG. 11 is a view illustrating a relationship between a magnetic fieldand an electric field applied to a brain of an object;

FIG. 12 is a view illustrating information obtained by visualizing amagnetic vector potential according to the type of a treatment coil;

FIG. 13 is a view illustrating an example of a method of calculating alocation and a direction of a coil;

FIG. 14 is a view illustrating examples of visualizing states in whichan electrical stimulus induced from a magnetic field of a treatment coilpropagates in a brain of an object are illustrated;

FIG. 15 is a flowchart illustrating a patch guide method according to anembodiment;

FIG. 16 is a view illustrating a result of simulating an electricalstimulation result according to an embodiment;

FIG. 17 is a view illustrating an embodiment of a method of matching animage;

FIG. 18 is a view illustrating an example of a 3-dimensional scan modelacquired by using a depth camera;

FIG. 19 is a view illustrating an example of photographing the head ofan object by a computing device, to which a camera module is connected,and guiding a location at which a patch is attached to the photographedhead of the object; and

FIG. 20 is a view illustrating a portable computing device and a depthcamera module connected to the portable computing device.

DETAILED DESCRIPTION

The above and other aspects, features and advantages of the inventionwill become apparent from the following description of the followingembodiments given in conjunction with the accompanying drawings.However, the inventive concept is not limited to the embodimentsdisclosed below, but may be implemented in various forms. Theembodiments of the inventive concept are provided to make the disclosureof the inventive concept complete and fully inform those skilled in theart to which the inventive concept pertains of the scope of theinventive concept.

The terms used herein are provided to describe the embodiments but notto limit the inventive concept. In the specification, the singular formsinclude plural forms unless particularly mentioned. The terms“comprises” and/or “comprising” used herein does not exclude presence oraddition of one or more other elements, in addition to theaforementioned elements. Throughout the specification, the samereference numerals dente the same elements, and “and/or” includes therespective elements and all combinations of the elements. Although“first”, “second” and the like are used to describe various elements,the elements are not limited by the terms. The terms are used simply todistinguish one element from other elements. Accordingly, it is apparentthat a first element mentioned in the following may be a second elementwithout departing from the spirit of the inventive concept.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by thoseskilled in the art to which the inventive concept pertains. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the specification andrelevant art and should not be interpreted in an idealized or overlyformal sense unless expressly so defined herein.

The terms “unit” or “module” used herein mean a hardware element such asan FPGA or an ASIC, and the “unit” or “module” performs some functions.However, the “unit” or “module” is not limited to software or hardware.The “unit” or “module” may be constituted in a storage medium that mayperform addressing, and may be configured to reproduce one or moreprocessors. Accordingly, as an example, the “unit” or “module” includeselements such as software elements, object-oriented software elements,class elements, and task elements, processes, functions, attributes,procedures, sub-routines, segments of a program code, drivers, firmware,micro-codes, circuits, data, a database, data structures, tables,arrays, and parameters. The elements and the functions provided in the“units” or “modules” may be coupled to a smaller number of elements andthe “units” or “modules” or may be separated to additional elements and“units” or “modules”.

In the specification, an “object” may include a human being or ananimal, or a part of a human being or an animal. For example, the objectmay include an organ, such as a liver, a heart, a uterus, a brain, abreast, or an abdomen, or a vessel. Further, the “object” may include aphantom. The phantom refers to a material having a volume that isclosest to the density of a living thing and the volume of an atomicnumber, and may include a spherical phantom having a property that issimilar to the human body.

Further, in the specification, the “user” is a medical expert and may bea doctor, a nurse, a clinical expert, and a medical image expert, andmay be an engineer who repairs a medical device, but is not limitedthereto.

Further, in the specification, a “magnetic resonance (MR) image” refersto an image for an object acquired by using a nuclear magnetic resonanceprinciple.

Hereinafter, exemplary embodiments of the inventive concept will bedescribed in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart illustrating a method for generating a3-dimensional brain map according to an embodiment.

FIG. 1 illustrates operations of the method, which are performed by acomputer, in time series. In the specification, a computer is construedto include all computing devices including at least one processor.

In operation S110, the computer acquires a brain MRI image of an object.

In an embodiment, the computer may directly acquire a brain MRI image ofan object from an MRI image acquiring device, as a workstation connectedto the MRI image acquiring device.

Further, the computer may acquire a brain MRI image of an object from anexternal server or another computer.

In the disclosed embodiment, a brain MRI image of an object refers to anMRI image that is captured by photographing a head part including abrain of the object. That is, a brain MRI image of an object refers toan MRI image including the skull and the scalp of the object, as well asthe brain of the object.

In operation S120, the computer segments (divides) the brain MRI imageacquired in operation S110 into a plurality of areas.

In an embodiment, the computer segments the brain MRI image acquired inoperation S110 for respective sections of the brain MRI image. Althoughthe computer, for example, may segment the brain MRI image acquired inoperation S110 into sections corresponding to a white matter, a graymatter, cerebrospinal fluid, a skull, and scalp, the types of segmentingthe brain MRI image are not limited thereto.

In an embodiment, the computer acquires the segmented brain MRI image ofthe object by inputting the brain MRI image of the object to a modelthat is learned by using a plurality of processed brain MRI images.

In an embodiment, the processed brain MRI images are images in which aplurality of images included in the brain MRI image are labeled.Further, the learned model is a model that receives a brain MRI Imageand outputs the segmented brain MRI image.

In an embodiment, the learned model refers to a model that is learned byusing machine learning, and particularly may refer to a model that islearned by using deep learning.

In an embodiment, although the learned model may be a model includingone or more batch normalization layers, activation layers, andconvolution layers, the inventive concept is not limited thereto.

In an embodiment, the learned model may include a horizontal pipelineincluding a plurality of blocks that extract high level characteristicsfrom low level characteristics of the MRI image and a vertical pipelinethat collects the extracted characteristics from the horizontal pipelineto perform segmentation, in order to perform segmentation of the MRIimage, the screen quality of which is relatively low.

Referring to FIG. 3, result images (b) of brain MRI images 300 obtainedby performing segmentation on original brain MRI images (a) of the brainMRI images 300 are illustrated.

In an embodiment, the computer performs post-processing of thesegmentation result.

In an embodiment, the computer performs connected component-based noiserejection. The connected component-based noise rejection method is usedto improve the result of the performed segmentation by using aconvolution neutral network (CNN).

Referring to FIG. 4, an example of the connected component-based noiserejection method is illustrated.

The computer acquires an improved segmentation image 410 by removing theremaining components 402 except for a connected component that is thelargest chuck of the segmentation image 400.

In an embodiment, the computer performs hole rejection. The holerejection is used to remove a hole that is one of the errors of theconvolution neutral network-based segmentation.

Referring to FIG. 5, an example of a post-processing method using holerejection is illustrated.

The computer acquires an improved segmentation image 510 by removing atleast a portion of the hole 502 included in the segmentation image 500.

In operation S130, the computer generates a 3-dimensional brain image ofan object including a plurality of segmented areas by using thesegmented brain MRI image of the object.

Referring to FIG. 6, a 3-dimensional brain image 600 generated from thebrain MRI image of the object is illustrated.

Further, an example of generating a segmented 3-dimensional brain image610 of the object, from a segmented 2-dimensional bran MRI image of theobject is illustrated in FIG. 6.

In operation S140, the computer generates a 3-dimensional brain map thatmay simulate a process of delivering an electrical stimulus to a brainof an object, based on the properties of a plurality of images includedin the 3-dimensional brain image generated in operation S130.

A detailed method of generating a 3-dimensional brain map of the objectand performing simulation by using the generated brain map will bedescribed with reference to FIG. 2.

FIG. 2 is a flowchart illustrating a method for generating a3-dimensional brain map and performing simulation according to anembodiment.

The method of FIG. 2 corresponds to an embodiment of the method ofFIG. 1. Accordingly, the contents described in relation to FIG. 1 arealso applied to the method of FIG. 2 even though they correspond to thecontents omitted in relation to FIG. 2.

In operation S210, the computer generates a 3-dimensional solid imageincluding a plurality of meshes that may simulate a process ofdelivering an electrical stimulus to the brain of the object, by usingthe 3-dimensional brain image of the object.

In an embodiment, the computer generates a 3-dimensional solid imageincluding a plurality of surface meshes including triangles orquadrangles.

In an embodiment, the computer generates a 3-dimensional solid imageincluding a plurality of surface meshes including tetrahedrons oroctahedrons.

The type of the meshes constituting the 3-dimensional solid image may bedifferently set according to the purpose of simulation.

In operation S220, the computer acquires physical characteristics of theplurality of areas for simulating current flows according to anelectrical stimulus to the brain of the object.

In an embodiment, the physical characteristics acquired in operationS220 include at least one of isotropic electrical conductivities andanisotropic electrical conductivities of the plurality of segmentedareas.

In an embodiment, the isotropic electrical conductivities may beacquired by allocating electrical conductivities, which have been knownthrough experiments, to the respective segmented areas.

For example, the electrical conductivities known for the respectiveareas of the brain are as in Table 1.

TABLE 1 Area Electrical conductivity (S/m) White matter 0.126 Graymatter 0.276 Cerebrospinal fluid 1.65 Skull 0.01 Scalp 0.465

The anisotropic electrical conductivities realize the anisotropy of thewhite matter fibers in the white matter of the brain.

In an embodiment, the anisotropic electrical conductivity is acquiredfrom a conductive tensor image for the brain of the object.

For example, the computer acquires a conductive tensor image for thebrain of the object from the brain MRI image of the object, and acquiresthe anisotropic electrical conductivities of the plurality of segmentedareas by using the acquired conductive tensor image.

In another embodiment, the brain MRI image of the object includes adiffusion tensor image, and the computer acquires the anisotropicelectrical conductivities of the plurality of segmented areas by usingthe diffusion tensor image of the object.

Referring to FIG. 7, an example of the diffusion tensor image 700 isillustrated.

An eigenvector of the diffusion tensor image is known to coincide withan eigenvector of a conductive tensor, and the computer may acquire ananisotropic electrical conductivity according to a direction of aneutral fiber included in the diffusion tensor image. For example, thedirection of the neutral fiber has a high electrical conductivity and adirection that is perpendicular to the neutral fiber has a lowelectrical conductivity.

In operation S230, the computer simulates a state in which a specificelectrical stimulus propagates in the brain of the object when thespecific electrical stimulus is applied to one point of the head of theobject, by using a 3-dimensional brain map.

In an embodiment, the computer simulates a state in which the electricalstimulus propagates in the brain of the object by using the mesh imageacquired in operation S210 and the physical properties acquired inoperation S220.

Referring to FIG. 8, an example of a simulation result is illustrated.

An electrical stimulus that may be applied to the head of an object mayinclude at least one of a magnetic field, an electric field, and acurrent, and when a magnetic field is applied to the head of the object,a current induced by the magnetic field may propagate to the brain ofthe object.

In an embodiment, the computer acquires a target stimulus point, towhich an electrical stimulus is to be applied in the brain of theobject. The computer acquires a location, to which an electricalstimulus is to be applied to the head of the object, to apply anelectrical stimulus to the target stimulus point, by using a3-dimensional brain map of the object.

For example, the computer may acquire a recommended path for deliveringan electrical stimulus from the scalp of the object to the targetstimulus point by using the 3-dimensional brain map of the object, andmay acquire a location, at which an electrical stimulus is to be appliedto the head of the object, from the recommended path.

A method for calculating and providing a location and a direction forapplying an electrical stimulus to the brain of the object by thecomputer will be described below.

FIG. 9 is a flowchart illustrating a transcranial magnetic stimulation(TMS) stimulus navigation method according to an embodiment.

FIG. 9 illustrates operations of the TMS stimulus navigation method,which are performed by a computer, in time series.

Referring to FIG. 10, an example of a TMS treatment method isillustrated.

The TMS is a treatment method that stimulates a specific portion of abrain of an object by using an electric field induced in the brain witha magnetic field generated by a treatment coil 1000, by allowing thecoil 1000 to approach one side surface of the head of the object 10.

The intensities and forms of the magnetic fields generated around thetreatment coil 100 will be different according to the shape of thetreatment coil 1000, and the forms in which the electrical signalspropagate will be different according to the forms of the head and thebrain of the object 10.

Accordingly, according to the disclosed embodiment, a stimulus pointbased on the type of the coil 1000 is calculated and provided, and asimulation result according to the forms of the head and the brain ofthe object 10 is provided.

In operation S910, the computer acquires a target stimulus point, towhich an electrical stimulus is to be applied in the brain of theobject.

The target stimulus point is selected based on a clinical or theoreticalbasis according to a disease that is to be treated. In an embodiment,the target stimulus point is indicated by using a 3-dimensional brainimage or a 3-dimensional brain map of the object generated by thedisclosed embodiment.

In operation S920, information on a spatial distribution of magneticvector potentials of the TMS treatment coil is acquired.

In an embodiment, the information on the spatial distribution includesinformation that is obtained by visualizing a magnetic vector potentialby using a magnetic dipole according to the shape of the treatment coil.

Referring to FIG. 12, information 1210 and 1260 that is obtained byvisualizing a magnetic vector potential according to the types of thetreatment coils 1200 and 1250.

In operation S930, the computer acquires one or more parameters foracquiring an optimal stimulus condition for the target stimulus pointacquired in operation S910 from the spatial distribution acquired inoperation S920.

In an embodiment, the optimal stimulus condition for the target stimuluspoint refers to a condition in which the intensity of the magnetic fieldapplied to the target stimulus point is maximized by the treatment coil.

Referring to FIG. 11, a relationship between the magnetic field and theelectric field applied to the brain of the object is illustrated.

Referring to simulation images (a) of a relationship diagram 1100 ofFIG. 11, images that are obtained by visualizing the intensity of amagnetic field applied to the brain of the object, the intensity of agradient (potential), and the intensity of an electric field induced bya magnetic field are illustrated. The intensity of the electric fieldapplied to the brain of the object may be calculated by adding themagnetic field and the gradient applied to the brain of the object.

Referring to a graph (b) of the relationship diagram 1100 of FIG. 11, acorrelation of a magnetic field applied to the brain of the object andan electric field induced by the magnetic field is illustrated.

According to the graph (b), it can be seen that a stronger electricfield is induced in the brain of the object as a stronger magnetic fieldis applied to the brain of the object.

Accordingly, it can be seen that the optimal stimulus condition for thetarget stimulus point is determined such that the intensity of themagnetic field applied to the target stimulus point by the treatmentcoil is maximized.

In an embodiment, the parameter acquired by the computer includes anoptimal point having the highest magnetic vector potential value in aspatial distribution of a magnetic vector potential induced by the coil.

Further, the parameter acquired by the computer includes an optimalvector that is a normal vector, of which a product of the normal vectorand the gradient is minimized, of the normal vectors, of which theoptimal points becomes start points.

Referring to FIG. 12, the optimal points 1212 and 1262 and the optimalvectors 1214 and 1264 of the magnetic vector potentials 1210 and 1250are illustrated.

The optimal point f and the optimal vector v are calculated by Equations1 and 2.

$\begin{matrix}{\max\limits_{x,y,z}\; {f( {x,y\;,z} )}} & \lbrack {{Equation}\mspace{14mu} 1} \rbrack \\{\min\limits_{i,j,k}\; {{\nabla{f( {\overset{\_}{x},\overset{\_}{y},\overset{\_}{z}} )}^{T}}{v( {i,j,k} )}}} & \lbrack {{Equation}\mspace{14mu} 2} \rbrack\end{matrix}$

In Equation 2, ∇f(x, y, z) are values that are obtained bydifferentiating f used when the optimal point is defined at x, y, z, andv(i, j, k) refer to normal vectors in the direction of (i, j, k).

In operation S940, a location and a direction of the coil that satisfiesan optimal stimulus condition for the target stimulus point acquired inoperation S910, by using the parameter acquired in operation S930.

In an embodiment, the operation of calculating the location and thedirection of the coil includes an operation of calculating the locationand the direction of the coil that allows the target stimulus point toapproach an optimal vector from an optimal point most.

Referring to FIG. 13, an example of a method for calculating a locationand a direction of the coil is illustrated.

If the object 10 and a target stimulus point (S, 12) of the object areacquired, the computer determines one point 14 on scalp that is closestfrom the target stimulus point 12.

Then, it is assumed that a distance between the target stimulus point 12and one point 14 on the scalp that is closest to the target stimuluspoint 12 is d and a vector, of which a start point is the point 14 andof which an end point 12 is the target stimulus point 12, is K. Further,a thickness of the coil 1310 is 2P.

The computer generates and applies a matrix as in Equation 3 in whichthe vector K 1320 and the optimal vector 1312 of the coil 1310 arearranged.

$\begin{matrix}{{{R_{x}(\theta)} = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos \; \theta} & {{- \sin}\; \theta} \\0 & {\sin \; \theta} & {\cos \; \theta}\end{bmatrix}}{{R_{y}(\theta)} = \begin{bmatrix}{\cos \; \theta} & 0 & {\sin \; \theta} \\0 & 1 & 0 \\{{- \sin}\; \theta} & 0 & {\cos \; \theta}\end{bmatrix}}{{R_{z}(\theta)} = \begin{bmatrix}{\cos \; \theta} & {{- \sin}\; \theta} & 0 \\{\sin \; \theta} & {\cos \; \theta} & 0 \\0 & 0 & 1\end{bmatrix}}} & \lbrack {{Equation}\mspace{14mu} 3} \rbrack\end{matrix}$

Accordingly, the location of the coil is calculated as in Equation 4.

LOCdipole=S+K*(D+P)   [Equation 4]

In operation S950, when the treatment coil is located at the locationcalculated in operation S940 in the direction calculated in operationS950, the computer simulates a state in which an electrical stimulusinduced from the magnetic field of the treatment coil propagates in thebrain of the object.

In an embodiment, the computer performs simulation by using the3-dimensional brain map generated according to the methods of FIGS. 1and 2.

For example, the computer may generate a brain MRI image of the object,and may generate a 3-dimensional brain map that may simulate a deliveryprocess of an electrical stimulus for the brain of the object based onthe properties of the plurality of areas included in the acquired brainMRI image.

The computer simulates a state in which an electrical stimulus by thecoil propagates in the brain of the object by using a 3-dimensionalbrain map.

Further, the 3-dimensional map may include a 3-dimensional solid imageincluding a plurality of meshes by which a delivery process of anelectrical stimulus for the brain of the object may be simulated.

In an embodiment, the computer visualizes a state in which an electricalstimulus induced from a magnetic field of a treatment coil by using a3-dimensional solid image.

Referring to FIG. 14, examples of visualizing states in which anelectrical stimulus induced from a magnetic field of a treatment coilpropagates in a brain of an object are illustrated.

In the disclosed embodiment, the computer is connected to a robot armdevice provided with a TMS treatment coil. The robot arm device includesa machine device that may move the TMS treatment coil to a locationdesignated by the computer.

The robot arm device may automatically perform a treatment using a TMScoil on a patient according to a calculation result of the computer bymoving the TMS treatment coil to a location designated by the computeraccording to the disclosed embodiment.

FIG. 15 is a flowchart illustrating a patch guide method according to anembodiment.

In the disclosed embodiment, a patch includes a brain stimulating patch.For example, the brain stimulating patch may include an electricalstimulation patch and an ultrasonic stimulation patch. Further, thepatch includes an EEG patch. Meanwhile, the types of the patchesaccording to the disclosed embodiment are not limited to theabove-mentioned examples.

In operation S1510, the computer acquires a 3-dimensional scan modelincluding the head of the object by using a depth camera.

The depth camera may include a 3-dimensional laser scanner of atriangulation method, a camera using a structural ray pattern, and adepth camera using a time-of-flight (TOF) method using a reflection timedifference of an infrared ray.

The depth camera is used to acquire a 3-dimensional scan model byreflecting distance information on an image.

In an embodiment, an object, that is, a patient is seated on a circularchair with no back, and a user, that is, a doctor locates the depthcamera such that the face of the patient is viewed well from a height ofthe face of the patient by using a temporary fixing device, such as, atripod.

The doctor starts scan by using the depth camera, and acquires a3-dimensional scan model including the head of the patient by rotatingthe patient slowly by one turn.

In an embodiment, the depth camera is provided in a fixing module thatmay be automatically rotated, and may acquire a 3-dimensional scan modelas the depth camera is rotated round the patient located at the center.

Meanwhile, according to the disclosed embodiment, in order that3-dimensional scan may be allowed without using separate high-pricedequipment, the depth camera module is connected to a portable computingdevice (for example, a smartphone or a tablet PC), the computing device,to which the camera module is connected, is fixed by using a temporaryfixing device, such as a tripod, which may be easily obtained, and a3-dimensional scan model may be acquired by seating the patient on astool and rotating the patient.

Referring to FIG. 20, a portable computing device 2000 and a depthcamera module 2010 connected to the portable computing device 2000 areillustrated.

Further, referring to FIG. 18, an example of a 3-dimensional scan model1800 acquired by using a depth camera is illustrated.

In an embodiment, the computer generates a 3-dimensional model includingthe head of an object by using a distance image collected by using adepth camera, and aligns images photographed at different time pointsand adds the images to reconstruct a 3-dimensional model of the object.For example, the computer reconstructs a model by collecting3-dimensional data in a point cloud form in distance images collected byusing the depth camera. However, the method for generating a3-dimensional model is not limited.

In operation S1520, the computer acquires a 3-dimensional brain MRImodel of the object.

In an embodiment, an operation of acquiring a 3-dimensional brain MRImodel of an object includes an operation of acquiring a brain MRI imageof an object, and an operation of generating a 3-dimensional brain mapof the object that may simulate a delivery process of an electricalstimulus for the brain of the object based on the properties of aplurality of areas included in the brain MRI Image of the object.

Further, the operation of generating a 3-dimensional brain map of theobject includes an operation of generating a 3-dimensional solid imageincluding a plurality of meshes, by which a delivery process of anelectrical stimulus for the brain of the object may be simulated.

The 3-dimensional brain map generating method described in relation toFIGS. 1 to 8 may be used as the method for acquiring a 3-dimensionalbrain MRI model of the object by the computer in operation S1520.

In operation S1530, the computer performs matching of the 3-dimensionalscan model including the head of the object and the brain MRI model ofthe object.

Referring to FIG. 17, an example of matching an image is illustrated.Referring to the image 1700 illustrated in FIG. 17, a brain MRI pictureof an object and an image obtained by modeling a brain structure of theobject overlap each other.

In the image 1700, the lower three images correspond to an example inwhich the brain MRI picture and the image obtained by modeling a brainstructure do not match with each other. Further, in the image 1700, theupper three images correspond to an example in which the brain MRIpicture and the image obtained by modeling a brain structure do notmatch with each other.

The computer calculates a change generated in the brain of the object byan electrical or ultrasonic stimulus of a patch according to a locationat which the patch is attached, by using a brain MRI model. Further, thecomputer calculates a location at which the patch is to be actuallyattached, by using a 3-dimensional scan model including the head of theobject.

Accordingly, the computer may calculate a location at which the patch isto be attached to the head of the object by matching the 3-dimensionalscan model including the head of the object and the brain MRI model ofthe object, and thus may calculate a change generated in the brain ofthe object. Similarly, the computer may calculate a location at whichthe patch is to be attached to the head of the object to cause aspecific change in the brain of the object, and may provide the result.

In an embodiment, the operation of performing matching by the computerincludes an operation of calculating a scan model and a facial featureof a brain MRI model, and an operation of matching the scan model andthe brain MRI model by using the scan model and the facial feature ofthe brain MRI model.

The scan model including the head of the object and the brain MRI modelof the object are different, and it is difficult to match them.Accordingly, the computer may match the two models by using the facialfeature of the object.

In an embodiment, the operation of calculating the facial feature of thescan model including the head of the object includes an operation ofacquiring a color image and a depth image including the head of theobject, an operation of calculating the facial feature of the object byusing a color image including the head of the object, and an operationof calculating a 3-dimensional location of the facial feature of theobject by using a depth image including the head of the object.

Referring to FIG. 18, an example of generating a model 1820 that isobtained by matching the scan model 1800 including the head of theobject and the brain MRI model 1810 of the object is illustrated.

In operation S1540, the computer acquires an image obtained byphotographing the head of the object by using the depth camera.

For example, the doctor may move the temporarily fixed depth camerawhile directly carrying the depth camera such that the depth camerafaces the head of the patient.

In operation S1550, one location of the image captured in operationS1540 and one location on the matched model are matched.

For example, when the computer photographs one point of the head of theobject by using the depth camera, it calculates to which portion on thematched model one point being photographed corresponds.

In an embodiment, the computer matches the captured image and thematched model, and displays an image that guides the location of thepatch, which is to be attached to the head of the object.

Referring to FIG. 19, the computing device 1900, to which the depthcamera module is connected, photographs the head 1910 of the object, andthe computing device 1900 displays an image that guides the location1920 for attaching the patch 1930 to the photographed head 1910 of thehead 1910.

In an embodiment, the computing device 1900 determines a location atwhich the patch 1930 is to be attached on the matched model, anddisplays a location 1920 corresponding to a location determined in thecaptured image.

Further, the computing device 1900 recognizes a patch 1930 in thecaptured image, and guides a movement direction of the recognized patch1930.

Further, the computing device 1900 determines whether the recognizedpatch 1930 has been attached to the determined location 1920.

In an embodiment, at least one marker is attached to or displayed on thepatch 1930. For example, at least one of a specific figure, a specificcolor, and a 2-dimensional code is attached to or displayed on the patch1930, and the computing device 1900 recognizes the patch by using themarker attached to or displayed on the patch 1930 and tracks themovement of the patch 1930.

For example, when the doctor photographs the head of the patient whilechanging the location of the head by using the computing device 1900 orthe depth camera connected to the computing device 1900, the location ofthe head of the patient displayed on the computing device 1900 is alsochanged and the location of the patch 1930 recognized by the computingdevice 1900 is also changed. In this case, the computing device 1900tracks the patch 1930 even when the computing device 1900 is moved sothat the doctor guides the patch 1930 such that the patch 1930 may beattached to an accurate location of the head of the patient.

In an embodiment, the computing device 1900 recognizes the patch 1930 inthe captured image, and guides a movement direction of the recognizedpatch 1930. For example, the computing device 1900 displays a movementdirection of the patch 1930 so that the patch 1930 may be attached tothe determined location 1920.

Further, the computing device 1900 determines whether the recognizedpatch 1930 has been attached to the determined location 1920. Forexample, the computing device 1900 may determine whether the locationfinally recognized by the patch 1930 corresponds to the determinedlocation 1920, and when the determined location 1920 and the location atwhich the patch 1930 is attached are different, the computing device1900 may provide an alarm that requests change of the location of thepatch 1930.

In an embodiment, the computing device 1900 recognizes the patch 1930attached to the head of the object in the captured image, and guides thelocation of the recognized patch 1930. The computing device 1900acquires a location on the matched model, which corresponds to thedetermined location of the patch 1930.

For example, when an EEG brain wave inspection is performed, an EEGpatch is attached to a specific location regardless of the shape andstructure of the head of the user or an EEG patch is attached to anarbitrary location. In this case, it is difficult to know from whichdirection of the brain of the object the brain wave acquired by the EEGpatch has been received.

Accordingly, according to the disclosed embodiment, the computing device1900 photographs the head of the object, to which at least one EEG patchis attached, and acquire the locations of the one or more recognized EEGpatches from the captured image.

The computing device 1900 may acquire a location on the matched model ofthe object corresponding to the acquired location of the EEG patch, andmay determine from which portion of the bran of the object the brainwave acquired by the EEG patch attached to the head of the object hasbeen received. For example, the computing device 1900 may analyze signalsources of the brain waves received from the EEG patches by utilizingthe disclosed embodiment.

FIG. 16 is a view illustrating a result of simulating an electricalstimulation result according to an embodiment.

Referring to FIG. 16, a 3-dimensional model of the head 1600 of anobject and an embodiment in which the patch 1610 is attached to onelocation on the 3-dimensional model are illustrated.

When the patch 1610 is attached to one location of the 3-dimensionalmodel of the head 1600 of the object, the computer simulates a result inwhich the electrical stimulus by the patch 1610 is delivered to thebrain 1650.

In an embodiment, the computer acquires a 3-dimensional brain map forthe brain 1650 of the object, and determines the location of the patch1610, which is to be attached to the head of the object, by using the3-dimensional brain map.

In an embodiment, the operation of determining a location of the patchincludes an operation of acquiring an objective of using the patch 1610,an operation of simulating a process of delivering an electricalstimulus to the brain 1650 of the object according the location at whichthe patch 1610 is attached to the head 1600 of the object, and anoperation of determining the location of the patch 1610 by using theacquired objective and the simulation result.

When a specific stimulus is to be applied to the brain 1650 of theobject, the computer may determine a location of the patch 1610 at whicha specific stimulus may be applied to the brain 1650 of the object byusing the simulation result.

The computer may match the location of the patch 1610 determinedaccording to the embodiment illustrated in FIG. 16 with one point of thehead of the object photographed by using the depth camera, and maydisplay an image that guides the patch to the matched location.

The steps of a method or an algorithm that have been described inrelation to the embodiments of the inventive concept may be directlyimplemented by hardware, may be implemented by a software moduleexecuted by hardware, or may be implemented by a combination thereof.The software module may reside in a random access memory (RAM), a readonly memory (ROM), an erasable programmable ROM (EPROM), an electricallyerasable programmable ROM (EEPROM), a flash memory, a hard disk, adetachable disk, a CD-ROM, or a computer readable recording medium in anarbitrary form, which is well known in the art to which the inventiveconcept pertains.

According to the disclosed embodiment, the brain MRI image may beautomatically segmented in a short time by segmenting the brain MRIimage by utilizing a model learned in advance.

Accordingly, the 3-dimensional brain image may be acquired by any one ina short time in the medical field, and a simulation result, by which theeffect of an electrical stimulus to the brain of the object may bevisually identified, may be provided.

Further, by guiding the location of the patch by using head modeling andMRI modeling, the location of the patch in consideration of differentheads and brain structures of people may be guided.

Further, a more accurate result may be obtained through the EEG brainwave inspection by determining the location of the EEG patch by usinghead modeling and MRI modeling.

The aspect of the inventive concept is not limited thereto, and otherunmentioned aspects of the inventive concept may be clearly appreciatedby those skilled in the art from the following descriptions.

Although the exemplary embodiments of the inventive concept have beendescribed with reference to the accompanying drawings, it will beunderstood by those skilled in the art to which the inventive conceptpertains that the inventive concept can be carried out in other detailedforms without changing the technical spirits and essential featuresthereof. Therefore, the above-described embodiments are exemplary in allaspects, and should be construed not to be restrictive.

What is claimed is:
 1. A patch guide method comprising: acquiring a3-dimensional scan model including the head of an object by using adepth camera, by a computer; acquiring a 3-dimensional brain MRI modelof the object; matching the scan model and the brain MRI model toacquire a matched model; acquiring an image captured by photographingthe head of the object by using the depth camera; and matching onelocation of the captured image and one location on the matched model. 2.The patch guide method of claim 1, further comprising: recognizing atleast one patch included in the captured image; determining a locationof the recognized patch in the captured image; and acquiring a locationof the matched model corresponding to the determined location of thepatch.
 3. The patch guide method of claim 1, wherein the matching of thescan model and the brain MRI model includes: calculating facial featuresof the scan model and the brain MRI model; and matching the scan modeland the brain MRI model by using the facial features of the scan modeland the brain MRI model.
 4. The patch guide method of claim 3, whereinthe calculating of the facial feature of the scan model includes:acquiring a color image and a depth image including the head of theobject; calculating the facial feature of the object by using the colorimage; and calculating a 3-dimensional location of the facial feature ofthe object by using the depth image.
 5. The patch guide method of claim1, further comprising: displaying an image that guides a location of thepatch, at which the head of the object is to be attached, in thecaptured image, wherein the displaying of the image includes:determining a location, at which the patch is to be attached, on thematched model; and displaying a location corresponding to the determinedimage in the captured image.
 6. The patch guide method of claim 5,further comprising: recognizing a patch in the captured image; guiding amovement direction of the recognized patch; and determining whether therecognized patch is attached at the determined location.
 7. The patchguide method of claim 1, wherein the acquiring of the 3-dimensionalbrain MRI model of the object includes: acquiring a brain MRI image ofthe object; and generating a 3-dimensional brain map of the object, bywhich a delivery process of an electrical stimulus for the brain of theobject is simulated, based on properties of a plurality of areasincluded in the brain MRI image, and wherein the patch guide methodincludes: determining a location of the patch, at which the head of theobject is to be attached, by using the 3-dimensional brain map.
 8. Thepatch guide method of claim 7, wherein the generating of the3-dimensional brain map includes: generating a 3-dimensional solid imageconstituted by a plurality of meshes, by which a delivery process of anelectrical stimulus for the brain of the object is simulated.
 9. Thepatch guide method of claim 7, wherein the determining of the locationof the patch includes: acquiring an object of using the patch;simulating a process of delivering an electrical stimulus to the brainof the object according to a location, at which the patch is attached tothe head of the object; and determining the location of the patch byusing the acquired objective and the simulation result.
 10. A computerreadable computer program recorded in a recording medium, which iscoupled to a computer that is hardware to perform the method of claim 1.